Subject Searching in Online Catalogs: Metaknowledge Used by Experienced Searchers

This paper begins to identify and characterize the knowledge used by experienced librarians while searching for subject information in online catalogs. Ten experienced librarians performed the same set of six subject searches in an online catalog. Investigated was the knowledge used to solve retrieval problems. This knowledge represents expertise in the use of the catalog. Data were collected through the use of think-aloud protocols, transaction logs, and structured interviews. Knowledge was defined as knowledge of objects (factual knowledge), knowledge of events (experiential knowledge), knowledge of performance (process knowledge), and metaknowledge. Metaknowledge is the sense of whole derived from the integration of factual, process, and experiential knowledge about the search and the conditions under which it is performed. The focus of this paper is on metaknowledge. For evidence of metaknowledge the data were examined for explanations that participants gave for their actions and observations, and for ways that participants evaluated their own progress during the process of searching. Reasons and explanations given by searchers were related to all phases of the library information retrieval process from the user's receipt of material to policies for collection development, and not just events directly related to the performance of a particular search task. Introduction While prior studies of online catalogs have investigated searching behavior of library users (Borgman, 1986; Hancock, 1987; Markey, 1983; Matthews, Lawrence, & Ferguson 1983), this article begins to identify and characterize the knowledge used by experienced librarians while subject searching in online catalogs. This was considered an important topic for investigation because online catalogs are difficult to use, and an understanding of the specialized knowledge used by librarians may facilitate the design of more usable systems. Catalogs are Difficult to Use There are a number of inherent characteristics of online catalogs that make them difficult tools to use, especially when someone is seeking subject information (Bates, 1972; Borgman, 1986; Connell, 1991; Matthews et al., 1983). In covering a library's collection, an online catalog includes information on a wide variety of disciplines and subject areas. This variety results in a vocabulary that is diffuse. The long history of library catalogs also affects vocabulary in that the records in online catalogs reflect not only current input standards, i.e., descriptive cataloging codes and subject cataloging traditions, but also standards devised for prior technologies and earlier times. Indirectly, catalog history affects the catalog because library collections (and therefore library catalogs) represent the output of recorded knowledge throughout the ages. Library collections reflect evolving human views of knowledge—the changing paradigms (in the sense of Kuhn, 1970) of the subject disciplines themselves. This means that even if there were no inaccuracies and inconsistencies in indexing over the years, paradigm shifts in subject disciplines would create variations in the vocabulary (controlled and natural language) used. All these factors make the catalog a complex tool for information access. Yet, despite these complexities we expect inexperienced users to search them independently. Many authors point out that serious problems can arise when a user does not have an accurate understanding ("mental model") of the limits and capabilities of a system (Borgman, 1983; Daniels, 1986; Dayton, Lundeen, & Pollock, 1980; Doszkocs, 1986; Hildreth, 1989). The importance of the user's model is obvious when considering the scope of the database. It does not make sense for a user to seek information about dinosaurs in a database devoted to cancer treatment; or, to look for journal articles in most contemporary library catalogs. Less obvious but equally important is the user's model of how the information in the database is indexed. One indexing concept used in catalogs is "specific entry." Specific entry is the subject cataloging guideline to enter a work under the subject heading that is coextensive with its subject coverage as distinguished from the broader, or more general heading that includes the subject of the book. Therefore, a book about jazz will be assigned the subject heading "Jazz", not "Music." Bates (1977) has shown that the awareness of the principle of specific entry influences significantly the success of a search in an online catalog. Specialist Knowledge The more one knows about a system the more effective he or she will be in using the system. Does the specialist knowledge that librarians have about library catalogs guide the ways they search those catalogs? Librarians build and provide the structure for the records in library catalogs and yet little research has been done about how librarians use these catalogs. Use studies involving library catalogs have concentrated on the end user. In contrast, it is the skilled user, the intermediary, that has been the focus of studies of bibliographic retrieval systems other than library catalogs. Bates' list of search tactics (Bates, 1979, p. 207), Fidel's "Selection Routine," (Fidel, 1988, p. 26) and Harter and Peters' list of search heuristics (Harter & Peters, 1985, p. 407) are descriptions of procedures for searching information retrieval systems that have resulted from such studies. The research presented here begins to identify and characterize the knowledge used by experienced librarians while searching for subject information in online catalogs. In order to give users more assistance in the use of our catalogs and guidance in choosing alternative approaches to information, it is first necessary to understand which approaches are successful and in what circumstances. If we begin to understand the knowledge that is used by skilled searchers of online catalogs then perhaps we can build that knowledge into our systems so that the infrequent user can benefit from specialist knowledge without having to develop the expertise explicitly. Method Ten experienced academic librarians performed the same set of six subject searches in the Illinet Online catalog used by libraries throughout the state of Illinois. Because of their experience, these librarians were considered to be experts in the use of the library catalog. Participants were chosen from two academic institutions in the state. Both catalogers and reference librarians were chosen as participants because each perform different phases of the information retrieval process. An effort was made to obtain participants who represented a range of years of experience. The years of experience for participants ranged from 12 to 33 years. Years of experience at their current institution ranged from 2 to 25 years. All catalogers had been employed full time as reference librarians for at least 2 years sometime during their career. One reference librarian had worked as a cataloger. The fact that some of the librarians had worked both as catalogers and as reference librarians was not seen as a disadvantage. The purpose of having both catalogers and reference librarians participate in the study was to provide the potential for variety in the knowledge used by the librarians in the study. Librarians who have worked both as catalogers and reference librarians may provide a different perspective than librarians who have had only one kind of experience. All the participants were experienced with computers. Most had used the online catalog since it first became available at their institution. The shortest period of time listed by any participant for using the system was 2 years. All participants had experience with other databases. Reference librarians used CD-ROMs for answering patron queries. Catalogers used the Online Computer Library Center (OCLC) for verifying and creating bibliographic records and in-house systems for administrative purposes. The knowledge investigated was the knowledge used to solve retrieval problems, independent of the particular subject area of the inquiry. This knowledge represents expertise in the use of the catalog. Therefore, techniques of knowledge acquisition, used by knowledge engineers in building expert systems, were the techniques used for knowledge elicitation. In an article on the quality of expertise, LaFrance (1989) observed that the major difference between expert knowledge and novice knowledge is not in quantity but in the structure and organization of the knowledge. Knowledge acquisition efforts must take into account not just what experts know, but how they organize what they know. Since any one knowledge elicitation methodology is not universally effective for all kinds of knowledge, using multiple methods to elicit knowledge from experts is more likely to result in an accurate and more complete representation of the expertise. Three methods were used to elicit knowledge from the librarians in the study; data were collected through the use of think-aloud protocols, transaction logs, and structured interviews. Think-Aloud Protocols The participants were instructed verbally and in writing to think aloud—to tell everything that passed through their minds as they thought about and performed each search. They were informed that their attempts and discarded ideas were as important as their final approach. These think-aloud protocols were audiotaped using a wireless lapel microphone clipped to the participant's clothing. This means of audiotaping enabled the participants to move around during the searching process and to use any auxiliary tools that they might normally use in conjunction with the online catalog. Typed, verbatim transcripts of the tapes were prepared. Think-aloud protocols do not provide a complete record of the searcher's thoughts because not everything is verbalized. In fact, there is evidence that the greater the experience with a task, the more automatic the process of performing the task. This means that intermediate steps in the process are carried out without conscious thought and, therefore, without verbalization (Ericsson & Simon, 1984). This is a disadvantage for knowledge elicitation because much of the structure of the expert's knowledge is not verbalized. However, what is verbalized represents the participant's thinking at the time. Think-aloud protocols are effective for revealing the development and progression of the expert's solution to the task at hand (process knowledge). Transaction Logs Computer transaction logs recorded the participants' interaction with the online catalog. Illinet Online appears as a single system for the user but it is actually two separate systems: the Library Computer System (LCS) and the Full Bibliographic Record (FBR). LCS is a short record circulation system. FBR contains two files, the Bibliographic File (with complete cataloging records) and the Authority File. The commands entered by participants into both systems were gathered and merged in the proper sequence to create a single file. Activating this file from a personal computer recreated the entire search session, commands, and system responses (what the participant saw on the screen), into a single, complete, transaction log. Once the complete transaction log had been created and printed, the commands were numbered consecutively. The commands were then matched against the audiotape of the corresponding think-aloud protocol to determine when each command occurred. The occurrence of each command was noted on the typed transcript of the verbal protocol. Transaction logs provide a bridge between a participant's think-aloud protocols and accompanying actions. They add the potential for increased understanding of the think-aloud protocols because the researcher has access to both searcher keystrokes and system responses. Structured Interviews Comparative studies of knowledge elicitation techniques show that the structured interview performs well for eliciting structure and process (Breuker & Wielinga, 1987). The sources of questions for the structured interview were the verbal protocol transcript and the complete transaction log. Because of the time required to obtain the protocol transcript and to create a complete transaction log, the interviews took place 1 week after the initial search session. The time lapse between the search session and the follow-up structured interview influenced the kinds of questions that could be asked. The time lapse was too great to obtain reliable data on the details of the participant's thinking behind the processes used, but clarification of broad process issues was possible. The structured interview concentrated on clarifying statements and actions made by participants that were not understood by the researcher, on establishing relationships among terms and processes used by the participant, and on the elaboration of allusions to previous search events. The structured interviews were audiotaped and transcribed for analysis. Document sources were used for explanation and/or clarification of data gathered by thinkaloud protocols, transaction logs, and structured interviews. Document sources included the Library of Congress Subject Headings (LCSH; 1989) the LC Subject Cataloging Manual: Subject Headings (SCM:SH; 1990), and descriptions of and instructions for using Illinet Online. Search Tasks In two searches the participants were asked to find material with similar subject coverage as the item described in an abstract. This kind of search task imitates a student's request for a book "like" a faculty-recommended book that is unavailable. The remaining four search tasks evolved from patron reference queries. The queries came from patrons' questions in two departmental libraries at the University of Illinois at Urbana-Champaign. The queries were in final negotiated form; i.e., the queries were in the form that the reference librarian searched after interviewing the patron. Five of the search tasks were selective: Participants were asked to find one or two good items that could help the patron answer the question asked. One was comprehensive: Participants were requested to find everything published during the last 5 years that was related to the topic. Questions for searching were chosen from the disciplines of economics and psychology. Bates (1972), in her dissertation, observed that because of its geographical emphasis, economics was a more difficult field for students to match subject headings than was the field of psychology. Difficulties were particularly apparent in the application of the principle of specific entry for subject assignment. By comparison, psychology headings were straightforward. The two abstracts used in this study were taken from Bates' dissertation. Using questions that posed difficulties for students could reveal how experienced librarians use their expertise to solve problems. As well, it was assumed that economics and psychology, because of their relationships to many other disciplines, are two fields in which most librarians have had some experience searching. Data gathering took place in the participants' own libraries. It was hoped that the research would take place in their own work area because then participants would have available any auxiliary tools that they might normally use in conjunction with the online catalog. Unfortunately, for a variety of reasons, catalogers were unable to perform the searches in their